…ian` (#5216)

Branching: #5269 >
#5216 >
#5322 >
#5335

The basic idea is to have a clone of `qml.Hamiltonian` with a better
name, `LinearCombination` - that inherits from `CompositeOp` and plays
nice with new opmath. The motivation is that sometimes it is still handy
to have an operator class for which you _know_ there is no funny nesting
and that has `coeffs` and `ops` separately on demand without processing.

ToDo

- [x] basic init
- [x] `tests/ops/op_math/test_linear_combination.py` pass
- [x] update map_wires
- [x] all tests pass
- [x] add pauli_rep
- [x] add operands attribute and make iterable (?)
- [x] update string repr
- [x] upgrade to internally use new opmath
- [x] update tests with default new opmath
- [x] dunder math methods support with new opmath
- [x] simplify and speed-up `simplify()`, also make sure to not act
in-place
- [x] Could we make the matrix generation workflow more general like
Sum?
- [x] Also, deferring to the pauli rep method for matrix generation
leads to substantial performance improvements.
- [x] toggle `__use_new_opmath`
- [x] tests_linear_combination pass
- [x] Integration tests
- [x] get all linear combination tests to pass locally
- [x] utilize super().sparse_matrix
- [x] use grouping functionality from Sum that Mudit is adding in
#5179
- [x] update matmul to handle other LinearCombination instances
- [x] remove top level import
- [x] remove unnecessary copies
- [x] add diagonalizing gates
- [x] bugfix trivial case of simplify empty LinearCombination
- [x] add trivial case for is_hermitian
- [ ] torch differentiation with simplify=True in constructor
- [x] changelog

Optional features (likely to be included in a follow-up)
- [x] `qml.Hamiltonian` points to either old Hamiltonian or
LinearCombination depending on `__use_new_opmath`
- [x] support Hermitian
- [ ] matrix method (optional)
- [ ] adjoint method (optional)
- [ ] batching support (optional)
- [ ] qml.dot points to LinearCombination
- [ ] take care of xfails (mostly about raiseing errors in other parts
of the codebase)
- [ ] update logic of adjoint differentiation to catch attempt to
differentiate lincomb coeffs
- [x] support in qml.equal
- [x]  add special matmul with other product or non-composite ops
- [x] add special matmul with other LinearCombination


[sc-56704]

---------

Co-authored-by: lillian542 <38584660+lillian542@users.noreply.github.com>
Co-authored-by: Astral Cai <astral.cai@xanadu.ai>
Co-authored-by: Mudit Pandey <mudit.pandey@xanadu.ai>
Co-authored-by: Christina Lee <christina@xanadu.ai>
Co-authored-by: Pietropaolo Frisoni <pietropaolo.frisoni@xanadu.ai>
Co-authored-by: albi3ro <chrissie.c.l@gmail.com>
Co-authored-by: Utkarsh <utkarshazad98@gmail.com>
Co-authored-by: lillian542 <Lillian.frederiksen@xanadu.ai>
Co-authored-by: Alex Preciado <alex.preciado@xanadu.ai>
Co-authored-by: Thomas R. Bromley <49409390+trbromley@users.noreply.github.com>
Co-authored-by: Vincent Michaud-Rioux <vincentm@nanoacademic.com>
Co-authored-by: Josh Izaac <josh146@gmail.com>
Co-authored-by: Nathan Killoran <co9olguy@users.noreply.github.com>
Co-authored-by: Matthew Silverman <matthews@xanadu.ai>
Co-authored-by: Mikhail Andrenkov <mikhail@xanadu.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

PR to toggle the `__use_new_opmath` and make it default.
This will require several changes in the codebase (and further updates
to demos, dataset and plugins).
Bigger changes should be offloaded in separate PRs. Small updates can be
gathered here.

Branching: #5269 >
#5216 >
#5322 >
#5335

- [ ] Make sure lightning works

```python
import os, sys
os.environ["OMP_NUM_THREADS"]="8"
import pennylane as qml
from pennylane import numpy as np
from timeit import default_timer as timer

if __name__ == "__main__":
    rng = np.random.default_rng(seed=1337)  # make the results reproducable
    mol = qml.data.load("qchem", molname="H2O", bondlength=0.958, basis="STO-3G")[0]

    hf_state = mol.hf_state
    ham = mol.hamiltonian
    wires = ham.wires
    dev = qml.device("lightning.qubit", wires=wires, batch_obs=True)

    n_electrons = mol.molecule.n_electrons
    singles, doubles = qml.qchem.excitations(n_electrons, len(wires))
    hf_state.requires_grad = False
    hf_ops = []

    for idx,i in enumerate(hf_state):
        if i==1:
            hf_ops.append((qml.PauliX, idx))

    @qml.qnode(dev, diff_method="adjoint")
    def cost(weights):
        for (x,i) in hf_ops: # HF state
            x(i)
        for idx,s in enumerate(singles):
            qml.SingleExcitation(weights[idx], wires=s)
        for idx,d in enumerate(doubles):
            qml.DoubleExcitation(weights[idx+len(singles)], wires=d)
        return qml.expval(ham)

    params = qml.numpy.array(rng.normal(0, np.pi, len(singles) + len(doubles)))
    opt = qml.GradientDescentOptimizer(stepsize=0.5)

    # store the values of the circuit parameter
    angle = [params]
    max_iterations = 50
    procs = int(os.getenv("PL_FWD_BATCH", "0"))
    pre_s = f"qubits={len(wires)},num_terms={len(ham.terms()[0])},procs={procs},"

    energies = []

    for n in range(max_iterations):
        start_grad = timer()
        params, prev_energy = opt.step_and_cost(cost, params)
        energies.append(prev_energy)
        end_grad = timer()
        angle.append(params)
        print(f"{pre_s},Step={n},Time_grad={end_grad-start_grad}")

    start_fwd = timer()
    energy = cost(params)
    energies.append(energy)
    end_fwd = timer()
    print(f"Energies={energies},Time_fwd={end_fwd - start_fwd}")
```

---------

Co-authored-by: lillian542 <38584660+lillian542@users.noreply.github.com>
Co-authored-by: Astral Cai <astral.cai@xanadu.ai>
Co-authored-by: Mudit Pandey <mudit.pandey@xanadu.ai>
Co-authored-by: Christina Lee <christina@xanadu.ai>
Co-authored-by: Pietropaolo Frisoni <pietropaolo.frisoni@xanadu.ai>
Co-authored-by: albi3ro <chrissie.c.l@gmail.com>
Co-authored-by: Utkarsh <utkarshazad98@gmail.com>
Co-authored-by: lillian542 <Lillian.frederiksen@xanadu.ai>
Co-authored-by: Alex Preciado <alex.preciado@xanadu.ai>
Co-authored-by: Thomas R. Bromley <49409390+trbromley@users.noreply.github.com>
Co-authored-by: Vincent Michaud-Rioux <vincentm@nanoacademic.com>
Co-authored-by: Josh Izaac <josh146@gmail.com>
Co-authored-by: Nathan Killoran <co9olguy@users.noreply.github.com>
Co-authored-by: Matthew Silverman <matthews@xanadu.ai>
Co-authored-by: Mikhail Andrenkov <mikhail@xanadu.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>